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ABSTRACT 

/ The ability to correctly answer reading comprehension 

test items, without having read the accompanying reading passage, was 
compared for third grade learning disabled students and their peers 
from a regular classroom* In the first experiment, fourteen multiple 
choice items were selected from the Stanford Achievement Test* No 
reading passages were provided, but the items were clustered 
according to their association with a particular passage* The 
learning disabled students answered correctly only slightly more than 
predicted by chance. Scores of the non-learning disabled group were 
significantly higher. In order to address the suggestion that the 
poorer performance of learning disabled students was caused by their 
lesser ability to read the test questions, a second experiment was 
conducted. A different sample of students included learning disabled 
and average third grade students. The examiner read each of the 14 
test items aloud, and all students were given sufficient time to 
answer all items. The average students again had higher scores than 
the learning disabled group. The results suggested that learning 
disabled students may be less likely than their non-learning disabled 
peers to apply test-taking strategies to reading comprehension 
questions. (GDC) 
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Abstract 

Previous research has indicated that students in many cases can 
answer reading comprehension test questions correctly without 
having read the accompanying pasoage^ The present research com- 
pared, in two experiments, the ability of learning disabled (LD) 
students and more typical age peers to answer such reading compre- 
hension questions presented independently of reading passages. In 
Experiment 1, LD students scored appreciably lower under condi- 
tions resembling standardized administration procedures. In 
Experiment 2, readiny decoding ability was controlled tor; how- 
ever, the performance differential remained the same<» Results 
suggested a relative deficiency on the part of LD students with 
respect to reasoning strategies and test-taking skills. In addi- 
tion, the validity of some tests of "reading comprehension'' was 
di scussed* 
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Are Learning Disabled Students "Test-Wise?": 
An Inquiry into Reading Comprehension Test Items 
For many years, there has been some argument over what 
reading comprehension tests "really? measure (e.g., Thorndike, 
1973-1974). The most cor^monly observed standardized reading 
comprehension item format consists of a passage and a number of 
associated multiple choice questions. Reading and understanding 
the passage is assumed to be a necessary pre-condition to 
correctly answering the questions. After examining the 
literature, however, one is forced to question the assumption of 
questioh dependence on the stimulus passage. Preston (1964) found 
that college students were able to answer reading comprehension 
items with the passages blacked out at a rate significantly above 
chance- Tuinman (1973-1974) administered five major tests to 
9,451 elementary-level students under several conditions. 
Students in the no passage condition (relevant passage had been 
blacked out) on the average achieved only 3055 fewer correct 
answers than subjects in the passage-in condition. Similar 
results were obtained by Pyrczak (1972, 1974, 1975, 1976) and 
Bickley, Weaver, and Ford (1968). A follow-up study of passage 
independence by Lifsoa, Scruggs, and Bennion (1984) revealed that 
passage-independent items are still quite common in elementary 
level achievement tests. College undergraduates were able to 
answer 75%, or almost 12 of 16 tjuestions on the Stanford 
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Achievement Test, Level P-3, without reading the associated 
passages. This score is considerably above that expected by 
chance responding. 

Scruggs, Bennion. and Lifscn (in press) interviewed 
elementary age students regarding their responses on a reading 
comprehension test. They found that students often chose their 
answers based upon their own prior knowledge, rather than content 
of the reading passage. When students reported using such prior 
infSrmation, they answered correctly in over 60% of the cases. 

Reading comprehension items wnich are independent of the 
associated passage can be answered on the basis of the following: 
(a) general knowledge, (b) interrelatedness of the questions on a 
particular passage, and (c) faulty item construction, i.e., keyed 
option is twice as ]ong or more precisely stated (Pyrczak, 1975). 
In the first two cases, the presence of enough information in the 
question stem to identify the topic is an important factor (e.g., 
"Which of the following statements in NOT true of penguins?"). 
Such a stem may render a question answerable in terms of 
information already available to the examinee and provide clues to 
the answers of related questions about the same passage that lack 
such information in the stem ("This passage is about: a) birds of 
South America, *b) birds of the Antarctic .... etc."). These 
clues which individuals apply to a. testing situation to maximize 
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their scores correspond to Millman, Bishop, and EbeTs (1965) 
criteria of test-taking skills, or "test wiseness." 

While test constructors may be able to point to high validity 
coefficients for their reading comprehension tests and subtests, 
an Important question arises concerning whether a/M students are 
equally able to answer questions with the above mentioned 
characteristics without reading the passage* Are some groups of 
students at a relative advantage/ disadvantage in ability to answer 
these questions without reading the passage? To answer this 
question, a group of students classified as learning disabled (LD) 
and a group of regular classroom students were administered a 
selection of multiple choice reading comprehension questions with 
the relevant passages removed. The conditions of this experiment 
were meant to resemble those of a normal testing situation—i.e. , 
students were required to read the questions without assistance. 
This did not permit us to determine the extent to which any 
observed differences between the regular and LD students were due 
to reasoning or variations in general knowledge between the two 
groups or simply reflected a difference in reading ability* To 
address this issue, a second experiment was performed to see if 
similar differences could be found when word reading was 
controlled for. 
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Experiment 1 
Method 

Subjects and Materials 

Subjects consisted of 67 regular classroom and resource room 
third grade students selected from several elementary schools in a 
western rural area. Of these subjects, 52 were regular classroom 
students and 15 were classified as LD by P.L. 94-142 and local 
criteria, which included a 40% discrepancy between actual and 
expected performance in two areas of academic functioning. The 
average grade equivalent of the total reading score of the non-L'D 
students on the Comprehensive Test of Basic Skills (CTBS) was 3.4 
(SD=.8), while the average CTBS total reading score for the LD 
students was 2.1 (SD=.5). 

Fourteen multiple choice reading comprehension questions 
without the accompanying passages were selected for this task. 
Items were drawn from the Stanford Achievejnent Test, Level P-3, 
Form E (1982). Items had been chosen to represent questions 
thought by the author to be answerable in terms of: (a) the 
general knowledge of the test taker, and (b) the degree to which 
the interrelatedness of the items served as a cue to the answers. 
These items were taken from the Lifson et al. (1984) study, in 
which students' ability to answer these questions had been 
documented. The items were kept in clusters which belonged 
together in terms of association with a particular passage. 
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Procedure 

Treatment was administered in regular instructional 
groupings. Materials were passed out and all students were told 
that they were about to take a reading test for which they would 
not be shown the accompanying reading passages, but that they 
should try their best to answer all questions. No time limit was 
imposed upon the task. 
Results 

The regular classroom group answered correctly approximately 
55% of the questions, f or ;iiean score of 7.8 (SD=1.96). This score 
was significantly above a chance score of 3.5 U(102) = 11.27, 
p<.001). In contrast, the LD students answered correctly only 35% 
of the questions, for a mean score of 4.9, only slightly higher 
than chance (t(28) = 1.77, ns). The obtained score of the non-LD 
group was significantly higher than the LD group (t(65) = 4.91, 
p<.001). 

Discussion 

The present findings suggest that regular classroom students 
are able to recognize and make use of cues in testing situations . 
in order to increase their scores, even when reading passages are 
deleted, and "reading comprehension" supposedly cannot be 
measured. Apparently, LD students are not able to benefit equ^iUy 
from these cues. Since neither group should have scored above 
chance on a reiding comprehension test with the reading passages 
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deleted, it is possible that a certain amount of bias exists 
against children with learning disabilities on some standardized 
tests of reading comprehension. Students in regular clsasses when 
unable to read or otherwise obtain meaning from reading passages 
are still able to answer correctly comprehension questionsV 
Students with learning disabilities, however, do not seem vp have 
these skills, and are thereby punished twice for a reading 
handicap: Once for being less able to read and comprehend the 
passage, and a second time for being unable to "second guess" test 
questions, as their nonhandi capped peers are apparently able to 
do. 

One possible explanation for this discrepancy between LD and 
regular classroom students is that LD students are simply less 
able to read (decode) the questions, and for that reason are less 
able to outguess the test. That is, LD students are less 
deficient in "test taking skills" than they are in reading 
ability. In order to address this question, a second experiment 
was designed, in which ability to read would be controlled for. 
Although the conditions in this experiment could not parallel 
those of standardized test procedures, they did allow for an 
assessment of the extent to which differential scores are 
attributable to generally lower reading skills. 
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Experiment 2 
Method 

Subjects and Materials 

The 42 subjects who participated in this investigation were 
different students drawn from the same population as those of 
Experiment 1, and consisted of 27 regular classroom third grade 
students and 15 third grade children classified as LD by P.L. 94- 
142 and local district criteria* Mean grade equivalent for the 
non-LD group (CTBS total reading) was 3.6 (SD=.9), and 1.9 (SD=.4) 
for the non-LD group. Materials were 14 items drawn from the 
Stanford Achievement Test, level P3, Form F, and were chosen on 
the same basis as those used in Experiment 1. Pages of the test 
were again left intact with questions left in the original order 
and the passages themselves blacked out during the copying 
process. 
Procedure 

Students were informed by their teacher that they were about 
to take a reading test without reading the corresponding passages. 
They were told to listen while the teacher read each item, and 
then answer the items. All students were given sufficient time to 
answer al 1 questions. 

Results and Discussion 
The students in regular classrooms answered correctly 65% of 
the fourteen items, for a mean score of 9.14 (SD=1.8). The LD 
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students, on the other hand, answered correctly only 45% of the 
items, for a mean score of 6.33 (SD=1.8). Although both obtained 
scores are well above chance, (t(52) = 12.02, and t^(28) = 4.325, 
ps<.001, for regular classroom and LD students, respectively), the 
regular classroom group maintained its advantage over the ID 
students, lt(40)=4.87, p<.001. The results suggest that learning, 
disabled students may be less likely to apply test-taking 
strategies to reading comprehension quc^stions to a degree of 
efficiency similar to their non-LD counterparts. 

General Discussion 
In Experiment 1, regular third grade classroom students were 
seen consistently to outscore their LD counterparts on a test of 
reading comprehension questions with corresponding passages 
deleted, and administered under conditions resembling standardized 
testing procedures. In Experiment 2, regular class third graders 
again outscored LD students, under conditions for which reading 
ability was controlled. The ability of third grade children in 
these cases to score 55% and 65% correctly on questions which 
refer to non-existent passages seems remarkable, ^nd brings into 
question the issue of what some testf of "reading comprehension" 
are really measuring. Such passage independent items have been 
thought to assess test-taking s .lis and in fact have been used 
as measures of "test-wiseness" (e.g., Derby, 1978). Although it 
is suggested that differences in the use of test-taking strategies 
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(such as use of prior knowledge, deductive reasoning, and 
elimination of implausable options) were responsible for much of 
the observed performance differences, other explanations are 
possible. Factors such as oral language decoding ability, 
attentional deficits, and test anxiety may have played a part in 
inhibiting performance on the part of the LD students. The role 
of these other factors in LD test performance is currently being 
investigated by the present authors (Scruggs, Bennion, & Lifson, 
1984; Taylor & Scruggs, 1983). Whatever such tests are seen to 
measure, however, it is clear that: (a) it is not "reading 

comprehension," and (b) children classified as LD are at an 

\ 

apparent disadvantage. 

An argument can be made that these comparisons are of trivial 
importance, since in standardized test administration, passages 
are not deleted; thcit all children in fact have equal access to 
passages which contain answers to reading comprehension questions. 
Although this argument has a certain face validity, some problems 
remain. First, since non-LD students can score so high on such 
items without reading the passages, the extent to which scores are 
a direct measure of "reading comprehension" seems uncertain. 
Second, since nearly all such tests are timed, students with 
.incomplete understanding of relevant passages but possessing an 
ability to "outguess" test questions under time constraints, 
clearly are at an advantage with respect to students not 
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possessing such an ability. In this case, differences in scores 
on reading comprehension tests may in fact reflect in part a bias 
toward students with superior ability to respond to specific cues 
in the test-taking situation. As has been seen in the present 
experiments, LD students may well find themselves on the negative 
side of any such bias. 

Two steps may be taken to help alleviate this potential 
source of bias. First, achievement tests should be revised so 
that reading comprehension tests directly assess comprehension of 
the provided passage. In fact, an informal review by the present 
authors of the major achievement tests indicates that many 
achievement test questions appear to be much l^ss "passage - * 
independent" since the work of Tuinman (1973-1974) and others of a 
decage ago. Second, it seems possible that at least some of these 
"test-taking skills" can be trained, and that thi| training may do 
much to correct this apparent disadvantage. The authors are at 
present investigating tne effectiveness of such training (Taylor & 
Scruggs, 1983). Although such improved scores on tests may not 
necessarily reflect increased achievement, these scores could 
reflect more accurately achievement gains st;udents have made', as 
evaluated by standardized achievement tests. 



ERIC 



13 



Reading Comprehension Tests 

13 

References 

Bickley, A. C, Weaver, W. W., & Ford, F. G. (1968). Information 
removed from multiple-choice item responses by selected 
grammatical categories. Psychological Reports . 23, 613-614. 

Derby, T. C. (1978). The effects of instruction in selected 
aspects of test-wiseness on the administration of standardized 
test items in the upper elementary schools . Unpublished 
doctoral dissertation. Southern Illinois University, 
Carbondale. 

Lifson, S. A., Scruggs, T. E. , & Bennion, K. E. (1984). Passage 

independence in reading achievement tests: A follow-up. 

Perceptual and Motor Skills . 58, 945-946. 
Preston, R. C. (1964). Ability of students to identify correct 

responses before reading. Journal of Educational Research . 58. 

181-183. 

Pyrcza'K, F. (1972). Objective evaluation of the quality of 

multiple-choice test items designed to measure comprehension of 
reading passages. Reading Research Quarterly . 8, 62-71. 

Pyrczak, F. (1974). Passage-dependence, of items designed to 
measure the ability to identify the main ideas of paragraphs: 
Implications for validity. Educational and Psychological 
Measurement , 34, 34-348. 



14 



I 

Reading Comprehension Tests 

14 

Pyrczak, F. (1975). Passage-dependence of reading comprehension 
questions: Examples. Journal of Reading , 1^, 308-311. 

Pyrczak, F. (1976). Context-indendence of items designed to 
measure the ability to derive the meanings of words from their 
context. Educational and Psychological Measurement , 36, 919- 
924. 

Scruggs, T. E., Bennion, K. E. , & Lifson, S. A. (1984, April). 

Spontaneously produced test-taking strategies of high and low 

comprehending elementary school children . Paper presented at 

the annual meeting of the American Educational Research 

Association, New Orleans. 
Scruggs, T. E., Bennion, K. E., & Lifson, S. A. (in press). An 

analysis of children's strategy use on reading achievement 

tests. Elementary School JournaK 
Tayi^**. C, i Scruggs, T. E. (1983), Improving the test-taking 

skills of learning disabled and behavioral ly disordered 

elementary school children. Exceptional Children , 50, 277. 
Thorndike, R. L. (1973-1974). Reading as reasoning. Reading 

Research Quarterly , 9^ 135-147. 
Tuinman, J. J. (1973-1974). Determining the passage dependency of 

comprehension questions in five major tests. Reading Research 

Quarterly , 9, 206-223. 



ERIC 



15 



Reading Comprehension Tests 

15 

Footnote 

Ijhis research was supported in part by a grant from the U.S. 
Department of Education. The authors would like to thank the 
excellent teachers of Cache Valley, Utah, for their assistance 
with this project: Marian Innocenti, Brenda Neiderhauser, Bonnie 
Olsen, Loila Anderson, and Edna Eams were particularly helpful. 
The authors would also like to thank Jill Barry, Ursula Pimentel, 
and Marilyn Tinnakul for their assistance in the preparation of 
this manuscript. Address requests for reprints to Thomas E. 
Scruggs, Exceptional Child Center, UMC 68, Utah State University, 
Logan, Utah 84322. 



16 



